Systems and methods for converting sensory data to haptic effects

ABSTRACT

A system includes a sensor, a processor configured to receive a signal output by the sensor, determine a position and descriptor of the sensor, determine a scene selection of a video, and create a haptic signal based on the selected scene, the signal output by the sensor, and the position and descriptor of the sensor, and a haptic output device configured to receive the haptic signal and generate a haptic effect based on the haptic signal. A system includes a sensor, a processor configured to receive a signal from the sensor, determine a type of event and activity associated with the sensor, and generate a haptic signal based on the signal from the sensor and the type of event and activity associated with the sensor, and a haptic output device configured to generate a haptic effect based on the haptic signal.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application claims the benefit of priority from U.S. ProvisionalPatent Application Ser. No. 61/968,799, filed Mar. 21, 2014, the entirecontent of which is incorporated herein by reference.

FIELD

The present invention is related to systems and methods for convertingsensory data to haptic effects, such as in sports media and usergenerated content media.

BACKGROUND

Adding compelling haptic effects at the right time in a video clip mayimprove the user's viewing experience and make the experience moreimmersive. This may be achieved by manually authoring the haptic effectsand adding the haptic effects in a timeline. However, manually authoringthe haptic effects can be time consuming for inexperienced users andeven if users are willing to spend enough time to create relevanteffects, they might not have the artistic touch that experienced hapticeffects authors have, which may result in bad effects that could ruinthe experience.

SUMMARY

It is desirable to provide systems and methods that automatically derivehaptic effects for a media piece of content for which one or moresensors has captured data related to the action in the content.

According to an aspect of the invention, there is provided a system thatincludes a sensor, a processor configured to receive a signal output bythe sensor, determine a position and descriptor of the sensor, determinea scene selection of a video, and create a haptic signal based on theselected scene, the signal output by the sensor, and the position anddescriptor of the sensor, and a haptic output device configured toreceive the haptic signal and generate a haptic effect based on thehaptic signal.

In an embodiment, the processor is further configured to determine adesired haptic effect and create the haptic signal based on the selectedscene, the signal output by the sensor, the position and descriptor ofthe sensor, and the desired haptic effect.

In an embodiment, the sensor is selected from the group consisting of:an accelerometer, a gyroscope, an inertial measurement unit, and acontact sensor.

In an embodiment, the system includes a user input device configured toallow a user of the system to indicate the position and/or descriptor ofthe sensor.

In an embodiment, the system includes a user input device configured toallow a user of the system to select the scene.

In an embodiment, the system includes a user input device configured toallow a user of the system to select the desired haptic effect.

In an embodiment, the processor is configured to synchronize a video ofthe selected scene with the haptic signal.

In an embodiment, the system includes a display configured to displaythe video of the selected scene synchronized with the haptic effectgenerated by the haptic output device.

In an embodiment, the system includes a plurality of sensors, and theprocessor is configured to receive signals output by the plurality ofsensors, determine positions and descriptors of the plurality ofsensors, and create the haptic signal based on the selected scene, thesignals output by the plurality of sensors, and the positions anddescriptors of the plurality of sensors.

According to an aspect of the invention, there is provided a method thatincludes receiving sensor data from a sensor, determining a position anddescriptor of the sensor, determining a scene selection of a video, andcreating a haptic signal based on the selected scene, the sensor data,and the position and descriptor of the sensor.

In an embodiment, the method includes synchronizing the selected sceneof the video and the haptic signal.

In an embodiment, the method includes receiving sensor data from aplurality of sensors, determining positions and descriptors of theplurality of sensors, and creating the haptic signal based on theselected scene, the sensor data, and the positions and descriptors ofthe plurality of sensors.

According to an aspect of the invention, there is provided a system thatincludes a sensor, a processor configured to receive a signal from thesensor, determine a type of event and activity associated with thesensor, and generate a haptic signal based on the signal from the sensorand the type of event and activity associated with the sensor, and ahaptic output device configured to generate a haptic effect based on thehaptic signal.

In an embodiment, the sensor is selected from the group consisting of:an accelerometer, a gyroscope, an inertial measurement unit, and acontact sensor.

In an embodiment, the type of event is selected from the groupconsisting of: a bump event, a swing event, an acceleration event, arotation event, a landing event, and a texture contact event.

In an embodiment, the activity is selected from the group consisting ofbiking, motocross, snowboarding, and skiing.

In an embodiment, the processor is further configured to synchronize avideo of the event with the haptic signal.

In an embodiment, the system includes a display configured to displaythe video of the event synchronized with the haptic effect generated bythe haptic output device.

In an embodiment, the system includes a user input device configured toprovide input to the processor to select the haptic effect to be outputby the haptic output device, and the haptic effect is selected from thegroup consisting of: a bump effect, a swing effect, a texture effect, anairtime effect, an acceleration envelope, and a rotation envelope.

According to an aspect of the invention, there is provided a method thatincludes receiving sensor data output by a sensor, selecting a type ofevent and activity associated with the sensor data, and generating ahaptic signal based on the sensor data, the type of event and theactivity.

In an embodiment, receiving sensor data output by the sensor includesreceiving sensor data output by a plurality of sensors.

In an embodiment, the method includes synchronizing a video of the eventand the haptic signal.

These and other aspects, features, and characteristics of the presentinvention, as well as the methods of operation and functions of therelated elements of structure and the combination of parts and economiesof manufacture, will become more apparent upon consideration of thefollowing description and the appended claims with reference to theaccompanying drawings, all of which form a part of this specification.It is to be expressly understood, however, that the drawings are for thepurpose of illustration and description only and are not intended as adefinition of the limits of the invention. As used in the specificationand in the claims, the singular form of “a”, “an”, and “the” includeplural referents unless the context clearly dictates otherwise.

BRIEF DESCRIPTION OF DRAWINGS

The components of the following Figures are illustrated to emphasize thegeneral principles of the present disclosure and are not necessarilydrawn to scale. Reference characters designating correspondingcomponents are repeated as necessary throughout the Figures for the sakeof consistency and clarity.

FIG. 1 is a schematic illustration of a system according to embodimentsof the invention;

FIG. 2 is a schematic illustration of a portion of the system of FIG. 1according to embodiments of the invention; and

FIG. 3 is a schematic illustration of a portion of the system of FIG. 1according to embodiments of the invention.

DETAILED DESCRIPTION

FIG. 1 illustrates a system 100 in accordance with an embodiment of theinvention. As illustrated, the system 100 includes one or more sensors102, which are configured to sense movement of an object and convert thesensed movement into sensor data, a video recorder 104 configured tocapture and record images, and an audio recorder 106 configured tocapture and record sound. In an embodiment, the sensor(s) 102, the videorecorder 104, and the audio recorder 106 may be part of the sameelectronic device. In an embodiment, the video recorder 104 and theaudio recorder 106 may be part of the same electronic device, and thesensor(s) 102 may be separate from the electronic device that includesthe video recorder 104 and the audio recorder 106. In an embodiment, thesensor(s) 102, the video recorder 104, and the audio recorder 106 may beseparate, stand-alone devices or part of separate, stand-alone devices.

A processor 110 is configured to process signals and data output by thesensor(s) 102, the video recorder 104, and the audio recorder 106, asdiscussed in further detail below. The system 100 also includes an inputtransformer 112, an output transformer 114, which may be part of theprocessor 110, and a decoder 116, which may also be part of theprocessor 110. Aspects of the input transformer 112, the outputtransformer 114, and the decoder 116 are discussed in further detailbelow.

As illustrated in FIG. 1, the system 100 also includes a haptic outputdevice 118 configured to output haptic effects to a user of the system,a display 120 configured to display images, such as the images capturedby the video recorder 104, and a speaker 122 configured to output sound,which may be the sound captured by the audio recorder 106. The hapticoutput device 118, the display 120, and the speaker 122 may be part ofan electronic playback device 130, as discussed in further detail below.In an embodiment, the haptic output device 118, the display 120, and thespeaker 122 may be separate devices that are configured to communicatewith each other through a wireless connection, for example. In anembodiment, the haptic output device 118 may be part of a wearabledevice, the display 120 may be part of a television, and the speaker 122may be a wireless speaker that is separate from the display 120.

The haptic output device 118 may include an actuator, for example, anelectromagnetic actuator such as an Eccentric Rotating Mass (“ERM”)actuator in which an eccentric mass is moved by a motor, a LinearResonant Actuator (“LRA”) in which a mass attached to a spring is drivenback and forth, or an actuator that includes a “smart material” such aspiezoelectric, electro-active polymers or shape memory alloys, amacro-composite fiber actuator, an electro-static actuator, anelectro-tactile actuator, and/or another type of actuator that providesa physical feedback such as a haptic (e.g., vibrotactile) feedback. Thehaptic output device 118 may include non-mechanical or non-vibratorydevices such as those that use electrostatic friction (ESF), ultrasonicsurface friction (USF), or those that induce acoustic radiation pressurewith an ultrasonic haptic transducer, or those that use a hapticsubstrate and a flexible or deformable surface, or those that provideprojected haptic output such as a puff of air using an air jet, and soon.

Electronic memory 124 may be used to store data sensed by the sensor(s)102, electronic memory 126 may be used to store data that is recorded bythe video recorder 104, and electronic memory 128 may be used to storedata that is recorded by the audio recorder 106. The memory 124, 126,128 may include one or more internally fixed storage units, removablestorage units, and/or remotely accessible storage units. The variousstorage units may include any combination of volatile memory andnon-volatile memory. The storage units may be configured to store anycombination of information, data, instructions, software code, etc. Inembodiments in which the sensor(s) 102, the video recorder 104, and theaudio recorder 106 are part of the same electronic device, the memory124, 126, 128 may be co-located. In embodiments in which the videorecorder 104 and the audio recorder 106 are part of the same electronicdevice, the memory 126, 128 may be co-located.

In an embodiment, a user may record video and/or audio of a scene orevent using the video recorder 104 and/or the audio recorder 106. In anembodiment, the video recorder 104 and the audio recorder 106 may bepart of the same recording device, such as a video camcorder, asmartphone, etc. The video and audio that is recorded may be stored inthe electronic memory 126, as discussed above. In an embodiment, thesensor(s) 102 may be placed on an object of interest, such as an objectbeing recorded in the video.

As discussed above, the data generated by the sensor(s) 102 may bestored in the electronic memory 124. In addition, the data generated bythe sensor(s) 102 may be transformed by the input transformer 112 priorto being stored in the electronic memory 124, as illustrated in FIG. 1.The transformation of the sensor data is considered to be an optionalstep and whether the transformation is needed may depend on the natureof the sensors being used. Details of embodiments of the sensor(s) 102are discussed in further detail below.

The decoder 116, which may be part of a media player configured toplayback the video, i.e. media file, is configured to read the datagenerated by the sensor(s) 102 from the electronic memory 124, andassociate the data temporally with the audio data and video data thatwere recorded and stored in the electronic memory 126. During mediaplayback, the decoder 116 may pass the sensor data through an outputtransformer 114 configured to transform the sensor data into a hapticoutput signal or haptic sensory commands, which include but are notlimited to, vibration, surface friction modulation, skin pinch, skinsqueeze, etc. The decoder 116 may be configured to synchronize thehaptic output signal that was transformed from the sensor data with thevideo data and the audio data so that the haptic effect is synchronizedwith the video and audio during playback. In an embodiment, thesynchronization may be completed by ensuring that time is the same inthe video data, the audio data, and the haptic effect during playback.

The system 100 may also include a user input device 140 that is insignal communication with the processor 110 and configured to allow auser to input data into the system. For example, the user may input databy selecting parameters related to the sensor(s) 102 and haptic effects,as described in further detail below. The user input device 140 mayinclude a display configured to display the parameters in a menu orother type of list format. In an embodiment, the user input device 140may be part of the playback device 130. In an embodiment, the user inputdevice 140 may be part of the video recorder 104. The illustratedembodiment is not intended to be limiting in any way.

The processor 110 may be a general-purpose or specific-purpose processoror microcontroller for managing or controlling the operations andfunctions of the system 100. For example, the processor 110 may bespecifically designed as an application-specific integrated circuit(“ASIC”) to control output signals to the haptic output device 118 toprovide haptic effects. The processor 110 may be configured to decide,based on predefined factors, what haptic effects are to be generated,the order in which the haptic effects are generated, and the magnitude,frequency, duration, and/or other parameters of the haptic effects. Theprocessor 110 may also be configured to provide streaming commands thatmay be used to drive the haptic output device 118 for providing aparticular haptic effect. In some embodiments, the processor 110 mayactually include a plurality of processors, each configured to performcertain functions within the system 100. The processor 110 may alsoinclude memory that includes one or more storage devices that mayinclude haptic effect profiles, instructions for how the haptic outputdevice 118 is to be driven, and/or other information for generatinghaptic effects. In an embodiment in which the entire system 100illustrated in FIG. 1 is part of a single electronic device, the memory124, 126 may be part of the processor 110.

The haptic output signal may then be transmitted from the processor 110,e.g., from the decoder 116 of the processor 110, to the haptic outputdevice 118 so that the person(s) experiencing the media through theelectronic playback device 130 that includes the haptic output device118 may more fully experience the event being played back. Theelectronic playback device 130 may be any device, such as an electronichandheld device, such as a mobile phone, gaming device, personal digitalassistant (“PDA”), portable e-mail device, portable Internet accessdevice, calculator, tablet, etc. The electronic playback device 130 mayinclude, but is not limited to, a handheld device with the display 120,which may be a high definition display, that displays the media, ahandheld object that is capable of producing haptic sensations oreffects, or an object attached to the user's body, leaning up to theuser's body, or otherwise able to transmit tactile sensations and hapticeffects to the user.

In an embodiment, the processor 110 and the haptic output device 118 maybe part of an electronic handheld device, which may be a phone or atablet, and the electronic handheld device may be configured to outputthe video data to a separate display 120, which may be a television.

In an embodiment, the system 100 may include a mobile phone having agyroscope, a compass, and three-axis accelerometer sensors for thesensors 102, as well as a built-in camera for the video recorder 104. Inthis instance, all of the components illustrated in FIG. 1, includingthe data recording sensor(s) 102, video recorder 104, audio recorder106, processor 110 including the decoder 116 and output transformer 114,haptic output device 118, display 120, speaker 122, input transformer112, and electronic memory 124, 126, 128 may be self-contained, and theentire system 100 may be affixed to the person or a piece of equipmentperforming the activity of interest.

In an embodiment, a first-person perspective video camera may be mountedto a helmet or piece of equipment performing the activity of interest,and the video camera may incorporate a number of data sensors 102, suchas accelerometers, a global positioning system (“GPS”), and gyroscopes,the input transformer 112, if needed, the electronic memory 124, thevideo recorder 104, the audio recorder 106, and the electronic memory126. The remaining parts of the system 100, such as the processor 110including the decoder 116 and output transformer 114, the haptic outputdevice 118, the display 120, and the speaker 122, may be located in aseparate playback device, such as the electronic playback device 130, asdiscussed above.

In an embodiment, the sensor(s) 102, which may include one or more datasensors, such as accelerometers, GPS, etc., may be affixed either to theperson or to equipment performing the activity of interest. Thesensor(s) 102 may be contained in a sensor box, or some other containerthat is configured to protect the sensor(s) 102. The sensor box may havedata recording means, such as the input transformer 112 and theelectronic memory 124, built-in, or may rely on a data connection tosecondary device (such as a mobile device) to record the data during theactivity.

In an embodiment, the sensor(s) 102 may be installed on a person who isthe subject of the video, and the video recorder 104, and optionally theaudio recorder 106, may be operated by another person. For example, theperson who is the subject of the video may be a biker, snowboarder orskier and at least one sensor 102 may be attached to the person's pieceof clothing or equipment, such as the snowboarder's boot, helmet, orsnowboard. The sensor(s) 102 may include accelerometers configured toprovide the g-force accelerations experienced by the snowboard,gyroscope sensors configured to provide the orientation of thesnowboard, and contact pressure sensors configured to provide the loadapplied to the snowboard by the snowboarder's boots. In an embodiment,the audio recorder 106 may be attached to an article of clothing orequipment, such as a helmet of the snowboarder so that the sound of thesnowboard passing over the snow may be captured.

In an embodiment, a recording session may be initiated in which thestream of acceleration data, gyroscope data, and contact pressure datais recorded alongside the video and audio data. The video recorder 104and/or audio recorder 106 may be mounted to the snowboarder's helmet,for example. In an embodiment, the video recorder 104 and/or audiorecorder 106 may be an array of video cameras mounted at variouslocations in a snow park's half pipe, where the snowboarding activity isto be performed. In an embodiment, the video recorder 104 may be anarray of video cameras located at various locations in a snow park'shalf pipe and the audio recorder 106 may be mounted to an article ofclothing or equipment of the snowboarder, as described above. Othercombinations of the locations of the sensor(s) 102, video recorder 104,and the audio recorder 106 may be used and may depend on the activitybeing recorded so that an optimum haptic/audio/visual experience may berealized by the user of the electronic playback device 130 duringplayback. The synchronization of all of these data streams may bemanaged by recording software, which may reside in the processor 110 ofthe system 100 illustrated in FIG. 1.

In an embodiment, flexible container formats, such as MPEG-4, that allowfor the storage of data other than video and audio in a single filecontainer, may be used. In such an embodiment, a particular set ofencoders may be needed to place the sensor data into the MPEG-4 fileduring recording. In an embodiment, special software may be written tostore the non-audio and video (A/V) sensor data in a separate file, butwith special markers in the sensor data to allow for propersynchronization at playback time. In this embodiment, very little inputtransformation may need to be applied, beyond shaping the sensor data toconform to the limitations of the designed recording format. The exactformat may be determined by the implementer. Once the snowboarder hascompleted his or her activity, the recording may be stopped. The MPEG-4file may be closed, and all of the sensor data may reside in the MPEG-4file.

In an embodiment, the playback device may be the electronic playbackdevice 130 of FIG. 1, and may be in the form of a mobile phone or tablethaving the display 120, the speaker 122, and a vibration device as thehaptic output device 118 to provide the haptic effect. In an embodiment,the playback device may be a gaming console connected to a televisionhaving the display 120 and the speaker 122, and also connected to agaming peripheral, such as a gamepad, that includes the haptic outputdevice 118 to provide the haptic effect.

Either at a later time, or concurrently with the activity beingperformed, one or more viewers may be interested in experiencing theactivity. To playback the activity, the viewer may launch theappropriate playback software on their playback device with theobjective of experiencing the performer's activity. In an embodiment,the playback software may include a player software application thatincorporates the sensor decoding scheme performed by the decoder 116, aswell as output transform software that may be run by the outputtransformer 114, in order to transform the sensor data into a hapticoutput signal suitable for the haptic output device 118 in the playbackdevice. In an embodiment, a player software application may incorporatethe sensor decoding scheme. The player software may rely on the outputtransform software being resident or otherwise pre-installed on theplayback device, and such output transform software may transform thesensor data into the haptic output signal suitable for the haptic outputdevice 118 in the playback device. In other words, the outputtransformer 114 and/or decoder 116 may be located on the playbackdevice.

In an embodiment, a player software application may rely on the playbackdevice's operating system software to perform the media playback, whichincorporates the sensor decoding scheme. The operating system softwaremay rely on the output transform software being resident or otherwisepre-installed on the playback device, such output transform softwaretransforming the sensor data into a haptic output signal suitable forthe haptic output device 118 in the playback device. The viewer may thenexperience haptic sensations associated with the viewing of theperformance, such haptic sensations being produced by the outputtransform software.

In an embodiment, the output transform software may include some of thesensor data, while other sensor data is ignored. For example, thegyroscope data may be included, while the contact pressure sensor datamay be ignored. The magnitudes of the acceleration data may be computedby computing the vector magnitude of X-Y-Z acceleration components. Thisacceleration magnitude signal may then be band-pass filtered such thatonly acceleration magnitude signals in the 20-200 Hz range are output,with other frequency content filtered out. The filtered accelerationsignal may then be put through a soft-knee compression algorithm togently clip the output magnitudes between +6 and −6 gees, then allnegative signals may be zeroed/ignored, and the resultant compressed,single-sided output signal may be used to control a magnitude ofvibration provided by the haptic output device between 0-100% vibration.In this way, the viewer may feel a representation of the accelerationvariants felt by the performer's snowboard.

In an embodiment, the acceleration data, gyroscope data, and contactpressure sensor data may be combined into a haptic output signal in thefollowing way. Whenever the contact pressure is low, the snowboarder maybe assumed to be jumping and therefore all haptic output is zeroed out.When the contact pressure is significant, the contact pressure may bemultiplied by the acceleration magnitude. This output may then bemultiplied by the change in orientation signal (the first timederivative of the orientation heading, derived from the gyroscopesignal) to obtain an “activity intensity” signal. This signal may thenbe filtered and compressed in such a way so as to obtain a reasonablerepresentation of the range of sensations experienced by the performerduring the snowboarding activity, and the signal may be applied to theplayback device's haptic output device 118.

In an embodiment, whenever the contact pressure is low, the snowboardermay be assumed to be jumping and therefore all haptic output is zeroedout. When the contact pressure is significant, the accelerationmagnitude may be band pass filtered in the 100-200 Hz range, forexample, and its output range may be highly compressed using a hard-kneecompressor. This signal may be applied to the playback device's hapticoutput device 118. The viewer holding or otherwise in contact with thehaptic output device 118 may then feel the sensations produced by theoutput transformer 114 and output by the haptic output device 118.

In accordance with embodiments of the invention, a plurality of sensors102 may be used as part of the system 100 described above. The sensors102 may be stand-alone sensors, such as accelerometers, gyroscopes,Inertial Measurement Units (“IMU”), contact sensors, etc., that may beattached to the subject of the event being recorded. For example thesensors 102 may be attached to the subject athlete's or user's body orapparel, such as a helmet, belt, shoe, boot, etc., or attached tohis/her sports equipment, such as a bike, snowboard, skis, etc. Thesensors 102 may be embedded in other wearable equipment carried by theuser during his/her ride or movement, such as a smartphone, actioncamera, smart watch, etc.

FIG. 2 is a schematic illustration of an implementation of theinvention. A first sensor 210 may be attached to, for example, a biker'shelmet, a second sensor 220 may be attached to, for example, a frontfork that supports a front wheel of the bike, a third sensor 230 may beattached to, for example, a main frame of the bike, and a fourth sensor240 may be attached to, for example, a rear fork that supports the rearwheel of the bike. The data captured by the sensors 210, 220, 230, 240may be stored or logged on the device or sensor, or streamed (forexample wirelessly) in real-time to a remote database 250, which may bepart of the processor 110 described above. The audio tracks capturedfrom different audio recorders 106 that are fixed in the environment mayalso be considered as sensory data that can be used to create thehaptics. A haptic track 260 that is generated may be embedded with thevideo or stored on a remote server to be streamed on the video playback.The haptic effects to be generated by the haptic output device 118 mayalso be created in real-time manner using the streamed data.

In an embodiment, the user may fix the different sensors 102 to his/herapparel and equipment in a way that will capture the most relevant data.The sensors' positions configuration may follow guidelines in accordancewith embodiments of the invention, or in another embodiment may be doneto fit the user's preferences (for more experienced users). Theguidelines are based on which sensor fits which situation the best. Forexample, if a snowboarder wishes to capture the snow texture and renderthe snow texture as an effect added to his/her movie, the guidelineswill tell the snowboarder to fix an accelerometer to his/her snowboard.

In order to synchronize all of the data channels and the media tracks,which include audio and video tracks, the user may have to perform aspecific signature movement, such as three successive jumps, in a way torecognize the signature movement in all the sensors', and video andaudio recorders' readings. The data synchronization is also possible bysynchronizing all of the internal clocks of the sensors to the sametiming, for sensors with internal clocks. After the sensor data andmedia content are acquired, the haptics track creation procedure maybegin in accordance with embodiments of the invention described below.

In an embodiment, the user may specify the type of activity, such asbiking, motocross, snowboarding, skiing, etc., then identify each sensorposition that is used, such as an ankle, a bike fork, etc., each sensortype, such as IMU, accelerometer, audio, etc., and sampling rate, suchas 200 Hz, via the user input device 140. In an embodiment, thepositions and types of sensors may also be selected by the user from amenu or list of preset values established using the system's sensorconfiguration general guidelines and presented to the user via the userinput device 140. In an embodiment, the type of sensor and sampling ratemay also be identified automatically by the system 100 when the sensoris connected to the system 100, which may be done by reading a specificheader embedded on the sensor that contains all of the importantdescriptors (i.e. “smart sensors”). The system 100 may then label eachsensor by a number and automatically label its acquired data. The system100 may also display a visual prototype of the equipment or apparel withan overlay of the positioning of the sensors and an illustration of thedifferent axes channel of the sensors via the user input device 140.

For a specific scene, which may be selected by the user via the userinput device 140, the user may select a specific type of haptic effectvia the user input device 140. To do so, the user may specify the sensorposition that he/she thinks will deliver the most relevant data to thescene, and also specifies the timings of the beginning and the end ofthe scene. By having the chosen sensor descriptors (type and position),the system 100 may suggest the different possible haptic effects (e.g.texture, bumps, signal envelope, etc.) that can be generatedautomatically from the data. The system 100 may also suggest onespecific effect when given a preset pattern. For some specific effects,the user may need to select a specific data channel from the sensor(e.g. acceleration on a specific axis in the case of a three-dimensionalaccelerometer), as described in further detail below.

For example, in a “biking” activity, if the user chooses the sensor“fork,” which is described as an accelerometer in the sensors table, thesystem may suggest “bump effects”, “envelope effect”, and “textureeffect” via the user input device 140. If the user picks the “bumpeffects,” the system 100 will ask the user which axis acceleration datato use (e.g., X axis) when deriving the haptics track 260. If a “textureeffect” is desired, the system 100 will use the three accelerationchannels available without asking the user for additional input. Theassociations between an activity, a sensor's position and type, and thepossible effects may be preprogrammed in the processor 110 of the system100 using expert knowledge. In an embodiment, the choice of the sensorto use in a scene may be done automatically by analyzing all the datarelated to the scene and choosing the channel that presents the mostactivity/entropy.

Given the type of the selected effect for the scene, the system 100 mayneed to use more information than from a single sensor in order tocreate a more compelling haptic effect. For example, when processinggyroscope data to create rotation effects, the system 100 may also usethe acceleration data captured from the same position as the gyroscope,if available. The created effect will then reflect if the object's orathlete's rotation is done in the air, which would be a less intenseeffect.

After establishing the list of all of the scenes to be haptified withthe desired sensors' information, the system 100 may retrieve therelevant sensory data from the database 250 and apply a specificalgorithm for each type of effect, described in further detail below,which results in a haptic effects signal per scene. After creating thehaptic effects for the different scenes, the system 100 concatenates thesimilar type effects in one track and then normalizes it by dividing itto its max value.

The different effects tracks may then be added to form the final haptictrack 260. Each effect track may also be added with a given weight(e.g., 0 to 1) to reflect the importance given by the user to eacheffect type. For example, if the user wants to promote bump effects andmake a texture effect more subtle, the user may give the first trackassociated with the bump effects a weight of 0.8 and a second trackassociated with the texture effect a weight of 0.5.

In an embodiment, the system 100 may create all of the possible haptictracks for each of the sensors and channels for the whole clip durationbased on the sensor's type and position. For example, for anaccelerometer, the system may create a track for “texture effect”,another track for “envelope effect” and a track per axis for the “bumpeffects.”

In an embodiment, the user may be presented with the different createdhaptic tracks in a video editing software framework, such as ProTools,and may have the choice to edit and cut each track to create the finalhaptic track for the clip. The user may also adjust and tune thedifferent tracks before mixing, as is done in audio editing. In anembodiment, the system 100 may analyze the created haptic tracks andassign each track to a part of the video when given the quantity ofinformation/entropy in each track at a certain time.

In order to create relevant haptic effects for a specific activity, suchas biking, motocross, snowboarding, skiing, etc., it is desirable todevelop a different algorithm for each type of sensor and for each typeof desired haptic effect, which will give the user more flexibility tochoose the type of haptic effects he/she finds more compelling for eachscene/clip. An effect may be associated with a specific event (hit,flip, etc.) in the scene/clip. The identification of the event in thesensor signals may be different, given the type of activity. Forexample, a bike hitting the ground creates a signature event in thethree-dimensional acceleration signals that is different from thesignature event created by a snowboard hitting the ice.

FIG. 3 illustrates a plurality of algorithms provided by embodiments ofthe invention that are used to adapt to different situations, asdescribed below. The algorithms may be programmed in one or more modulesin a processor, such as the processor 110 of the system 100 describedabove. The algorithms described herein may also be used in situations inwhich more than one sensor is used by using the methods described above.The following embodiments are provided as examples of differentalgorithms for different use cases and should not be considered to belimiting in any way.

For three-dimensional accelerations that may be sensed by anaccelerometer (or accelerometers) 310 located on the object of interest,such as a bike, different types of events and effects may bedistinguished and haptified. For example, a bump event, an accelerationenvelope, a swing event and/or a texture effect experienced by theobject of interest may be identified in the data signal output by theaccelerometer 310, and a corresponding algorithm may be applied to thedata signal to generate a haptic signal output to the haptic outputdevice 118 that corresponds to the event, as described below.

A “bump” event occurs, for example, when a bike contacts the ground withgreat force after a maneuver or a snowboarder contacts the snow withgreat force after a jump. In the bike (or motocross) case, such an eventis characterized by a strong peak (spike) in the vertical accelerationsignal sensed by an accelerometer attached to the bike. Bumps may alsoresult from sudden acceleration changes on any of the two other axes. Inthe case of snowboarding (or skiing), a bump event may be identified byspikes in both the vertical and the Medio-Lateral (ML) accelerationsignals, and a low acceleration signal on the Anterior-Posterior (AP)axis. Two bump effects algorithms 312 may be used to take this intoaccount.

A first algorithm, or so-called “simple bump” algorithm, is adapted forbike activity, motocross activity or any activity where a bump can beidentified from a single axis acceleration, namely when the absolutevalue of the signal is greater than a preset threshold value, forexample 5 g. The first algorithm uses only one axis acceleration signaland the user specifies the axis he/she is interested in haptifying. Thepreset value is adapted to the type of activity and to the position ofthe accelerometer on the apparel/equipment of the object of interest soas to integrate the possible signal attenuation.

The second algorithm, or so-called “three-axes bump” algorithm, uses thethree axes acceleration signals in order to identify the bumps. Thesecond algorithm starts by estimating the resultant acceleration valueof two of the axes. Whenever this estimated resultant acceleration valueis higher than a preset threshold, which is activity dependent, and thethird axis acceleration is lower than its own preset threshold, a bumpis spotted. The second algorithm is more suited for activities such asskiing or snowboarding.

Once the bump event is detected, the first and second bump effectsalgorithms 312 place a short (e.g., 60 ms) but full magnitude effect atthe same timing as the event in a haptic track 360.

An acceleration envelope algorithm 314 may be used to haptify differenttypes of events, such as an engine noise, small bumps experienced by abike, etc. The acceleration envelope algorithm 314 uses a single axisacceleration signal or a resultant of any combination of the three axessignals from the sensor 310 and derives haptic effects from the signalby calculating the acceleration envelope using root mean square (rms),mean, or maximum values for successive windows. For this algorithm, theuser may specify which of the three axes he/she wants to consider viathe user input device 140 described above. The envelope will becalculated only when the signal is greater than a preset threshold(noise threshold per activity) and smaller than the “bump” thresholddescribed above.

A swing event occurs in, for example, skiing and snowboarding activitieswhen the skier or snowboarder drifts fast on the Medio Lateral axis. Theswing event may be identified when the acceleration on the Medio Lateralaxis is higher than a specific threshold. A swing effect algorithm 316detects the event in the signal output from the sensor 310 andcalculates the acceleration envelope, as described above, around thisevent (e.g., for 1 s), then uses the envelope to generate acorresponding haptic effect.

A texture effect algorithm 318 may be used to render a texture effectusing accelerations along the three axes that are captured as close aspossible to the surface target. For example, the texture haptic effectfor snow is better derived using accelerations of the snowboard or skiitself. For a bike, it is typically better to derive the ground textureusing accelerations from the fork (non-attenuated accelerations signals)of the bike. The three accelerations signals may be transformed from thetime domain into the frequency domain using Fast Fourier Transform(“FFT”) and added as vectors in that domain to form a single signal. Theresultant signal may then be transferred back to the time domain, as isknown in the art. Therefore, details of the transformation are notprovided herein. The texture effect is generated by calculating theenvelope of the resultant signal, as described above with respect to theacceleration envelope.

For sensors that are in the form of a three-dimensional gyroscope 320, arotation envelope algorithm 322 may be used to haptify different type ofevents, such as a bike's handlebar rotations, a snowboard's 360° flips,a snowboard's sudden turns/stops, etc. The rotation envelope algorithm322 uses a single axis rotation speed signal and derives haptic effectsfrom the single axis rotation speed signal by calculating the signalenvelope using root mean square (rms), mean, or maximum values forsuccessive windows. For this algorithm, the user may specify which ofthe three axes he/she wants to consider. The envelope will be calculatedonly when the signal is bigger than a preset threshold (noise thresholdper activity). In a steering wheel or handlebar rotation case, a lowerthreshold will allow eliminating slower non-event rotations, while ahigher threshold implies haptifying only sudden fast movements that maycorrelate to “interesting events.”.

By embedding a contact sensor underneath or in an athlete's shoes,underneath a snowboard, or on a bike seat, events such as when anathlete is jumping in the air and when the athlete is landing again(e.g., for Parkour, snowboarding, motocross, etc.) may be detected. Thelanding event, which is characterized by a spike in the contact sensoroutput signal, may be then haptified by adding a short intense effect,such as for about 50 ms. Detecting the athlete's airtime may influencetuning the intensity of the other effects derived from the othersensors. For example, a rotation envelope will be of less intensity whenthe athlete is in the air.

Embodiments of the invention described above provide a methodology thathelps a user automatically derive haptic effects for a media piecehe/she has captured. Although the examples described above generallyfocus on sports media in which one or more sensors capture data relatedto the action in the media piece, embodiments of the invention are notso limited. The system described above allows for processing of all ofthe sensor data with no or very little intervention from the user, anddelivers a relevant haptics track that augments the media content andthus enhances the viewing experience.

Embodiments of the invention described above also provide a solution onhow to use different sensors in an optimal way and how to create hapticeffects from the combination of the different sensors at a certain time.Although the use of different sensors provides a better opportunity tocreate compelling haptics, the user may feel overwhelmed by the quantityof available data and may be confused on how to use the data at the sametime, for example, when to use what sensor(s) and what algorithm toapply, when deriving haptics. Embodiments of the invention provide asensor-to-haptics workflow when the athlete or equipment is equippedwith different sensors and provides the link between the captured dataand the haptic conversion algorithms, and thus optimize the procedure ofcreating haptics for user generated content from multiple sensors data.Embodiments of the invention guide the user through this procedure byusing guidelines and suggestions when needed and also fuses theinformation coming from different sensors at moments in order tooptimize the haptic effects creation.

Embodiments of the invention provide adapted algorithms for creatingdifferent haptic effects using sensor data in an effortless manner,which make the user's task easier. Embodiments of the invention allowthe user to use data describing the action in order to infer the hapticeffects, which allows the user to be sure that the haptic track createdis related to the actual action. The different algorithms describedabove allow the user to create different types of effects easily, andthen pick the effect he/she prefers.

The embodiments described herein represent a number of possibleimplementations and examples and are not intended to necessarily limitthe present disclosure to any specific embodiments. Instead, variousmodifications can be made to these embodiments as would be understood byone of ordinary skill in the art. Any such modifications are intended tobe included within the spirit and scope of the present disclosure andprotected by the following claims.

What is claimed is:
 1. A system comprising: a memory storing video dataand sensor data, the video data including an image associated with anobject, the sensor data measuring a movement of a sensor located on theobject, the sensor data being acquired concurrently with the video data;a processor, coupled to the memory, configured to: read the sensor datafrom the memory, the sensor data including a first acceleration signal,a second acceleration signal, a third acceleration signal, identify asensor type and a sensor position on the object, select an algorithm forgenerating a haptic effect, from a plurality of algorithms, based on thesensor type and the sensor position, each algorithm being configured togenerate a different haptic effect for a particular sensor type andsensor position, the plurality of algorithms including a three-axes bumpeffect algorithm having a first preset threshold value and a secondpreset threshold value, create a haptic signal based on the sensor dataand the selected algorithm, including: when the selected algorithm isthe three-axes bump effect algorithm: estimate a resultant accelerationvalue of the first and second acceleration signals, identify bump eventsin the sensor data, each bump event being identified when the estimatedresultant acceleration value is higher than the first preset thresholdvalue and the third acceleration signal is lower than the second presetthreshold value, and for each identified bump event, add a bump effectto the haptic signal, and transmit the haptic signal; and a hapticoutput device, coupled to the processor, configured to: receive thehaptic signal, and generate the haptic effect based on the hapticsignal.
 2. The system according to claim 1, wherein the sensor data isacquired from an accelerometer, a gyroscope, an inertial measurementunit, or a contact sensor, and the plurality of algorithms include asingle-axis bump effect algorithm, a swing effect algorithm, anacceleration envelope algorithm, a rotation envelope algorithm, alanding effect algorithm, or a texture effect algorithm.
 3. The systemaccording to claim 1, further comprising a user input device configuredto allow a user of the system to indicate the sensor type.
 4. The systemaccording to claim 1, further comprising a user input device configuredto allow a user of the system to select a scene within the video data.5. The system according to claim 1, further comprising a user inputdevice configured to allow a user of the system to select a desiredhaptic effect.
 6. The system according to claim 4, wherein the processoris further configured to synchronize a video of the selected scene withthe haptic signal.
 7. The system according to claim 6, furthercomprising a display configured to display the video of the selectedscene synchronized with the haptic effect generated by the haptic outputdevice.
 8. A method comprising: storing video data and sensor data in amemory, the video data including an image associated with an object, thesensor data measuring a movement of a sensor located on the object, thesensor data being acquired concurrently with the video data; reading thesensor data from the memory, the sensor data including a firstacceleration signal, a second acceleration signal, a third accelerationsignal; identifying a sensor type and a sensor position on the object;selecting an algorithm for generating a haptic effect, from a pluralityof algorithms, based on the sensor type and the sensor position, eachalgorithm being configured to generate a different haptic effect for aparticular sensor type and sensor position, the plurality of algorithmsincluding a three-axes bump effect algorithm having a first presetthreshold value and a second preset threshold value; creating a hapticsignal based on the sensor data and the selected algorithm, including:when the selected algorithm is the three-axes bump effect algorithm:estimating a resultant acceleration value of the first and secondacceleration signals, identifying bump events in the sensor data, eachbump event being identified when the estimated resultant accelerationvalue is higher than the first preset threshold value and the thirdacceleration signal is lower than the second preset threshold value, andfor each identified bump event, adding a bump effect to the hapticsignal; and transmitting the haptic signal to a haptic output devicethat generates the haptic effect based on the haptic signal.
 9. Themethod according to claim 8, further comprising synchronizing a selectedscene within the video data with the haptic signal.
 10. A systemcomprising: a memory storing video data and sensor data, the video dataincluding an image associated with an object during an event oractivity, the sensor data measuring a movement of at least one sensorlocated on the object, the sensor data being acquired concurrently withthe video data; a processor, coupled to the memory, configured to: readthe sensor data from the memory, the sensor data including a firstacceleration signal, a second acceleration signal, a third accelerationsignal, identify at least one of an event type or an activity type,select an algorithm for generating a haptic effect, from a plurality ofalgorithms, based on at least one of the event type or the activitytype, each algorithm being configured to generate a different hapticeffect for a particular event type or activity type, the plurality ofalgorithms including a three-axes bump effect algorithm having a firstpreset threshold value and a second preset threshold value, generate ahaptic signal based on the sensor data and the selected algorithm,including: when the selected algorithm is the three-axes bump effectalgorithm: estimate a resultant acceleration value of the first andsecond acceleration signals, identify bump events in the sensor data,each bump event being identified when the estimated resultantacceleration value is higher than the first preset threshold value andthe third acceleration signal is lower than the second preset thresholdvalue, and for each identified bump event, add a bump effect to thehaptic signal, and transmit the haptic signal; and a haptic outputdevice, coupled to the processor, configured to: receive the hapticsignal, and generate the haptic effect based on the haptic signal. 11.The system according to claim 10, wherein the sensor is anaccelerometer, a gyroscope, an inertial measurement unit, or a contactsensor.
 12. The system according to claim 10, wherein the event type isa bump event, a swing event, an acceleration event, a rotation event, alanding event, or a texture contact event, and the plurality ofalgorithms include a single-axis bump effect algorithm, a swing effectalgorithm, an acceleration envelope algorithm, a rotation envelopealgorithm, a landing effect algorithm, or a texture effect algorithm.13. The system according to claim 10, wherein the activity type isbiking, motocross, snowboarding, or skiing.
 14. The system according toclaim 10, wherein the processor is further configured to synchronize avideo of the event or activity with the haptic signal.
 15. The systemaccording to claim 14, further comprising a display configured todisplay the video of the event or activity synchronized with the hapticeffect generated by the haptic output device.
 16. The system accordingto claim 10, further comprising a user input device configured toprovide input to the processor to select the haptic effect to be outputby the haptic output device, the haptic effect including a bump effect,a swing effect, a texture effect, an airtime effect, an accelerationenvelope, or a rotation envelope.
 17. A method comprising: storing videodata and sensor data in a memory, the video data including an image ofan object during an event or activity, the sensor data measuring amovement of a sensor located on the object, the sensor data beingacquired concurrently with the video data; reading the sensor data fromthe memory, the sensor data including a first acceleration signal, asecond acceleration signal, a third acceleration signal; identifying atleast one of an event type or an activity type; selecting an algorithmfor generating a haptic effect, from a plurality of algorithms, based onat least one of the event type or the activity type, each algorithmbeing configured to generate a different haptic effect for a particularevent type or activity type, the plurality of algorithms including athree-axes bump effect algorithm having a first preset threshold valueand a second preset threshold value; generating a haptic signal based onthe sensor data and the selected algorithm, including: when the selectedalgorithm is the three-axes bump effect algorithm: estimating aresultant acceleration value of the first and second accelerationsignals, identifying bump events in the sensor data, each bump eventbeing identified when the estimated resultant acceleration value ishigher than the first preset threshold value and the third accelerationsignal is lower than the second preset threshold value, and for eachidentified bump event, adding a bump effect to the haptic signal; andtransmitting the haptic signal to a haptic output device that generatesthe haptic effect based on the haptic signal.
 18. The method accordingto claim 17, further comprising synchronizing a video of the event oractivity with the haptic signal.
 19. The method according to claim 8,wherein the sensor data is acquired from at least one of anaccelerometer, a gyroscope, an inertial measurement unit, or a contactsensor, and the plurality of algorithms include a single-axis bumpeffect algorithm, a swing effect algorithm, an acceleration envelopealgorithm, a rotation envelope algorithm, a landing effect algorithm, ora texture effect algorithm.
 20. The system according to claim 17,wherein the sensor data is acquired from at least one of anaccelerometer, a gyroscope, an inertial measurement unit, or a contactsensor.
 21. The system according to claim 17, wherein the event type isa bump event, a swing event, an acceleration event, a rotation event, alanding event, or a texture contact event, and the plurality ofalgorithms include a single-axis bump effect algorithm, a swing effectalgorithm, an acceleration envelope algorithm, a rotation envelopealgorithm, a landing effect algorithm, or a texture effect algorithm.22. The system according to claim 17, wherein the activity type isbiking, motocross, snowboarding, or skiing.